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Training in design and test development may not be of much help in dealing with 
the evaluation of Federally funded projects in a large city school system. Planning and 
carrying out evaluation in a large urban school system requires that much skill and 
time be spent on typically managerial pursuits. Therefore, a specialist in evaluation 
and research management is highly desirable. His duties should be mostly concerned 
with strategic issues such as management control, strategic planning, operational 
control, and the necessary person-to-person interactions. Such a person is likely to 
be less than adequate in dealing with technical evaluation issues. However, he should 
head up the evaluative effort, with a specialist in research design and test 
development assigned to carry out the actual evaluation. (TT) 
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Management Emphases in the Installation of an Evaluation Staff 
for Federally-Funded Projects in a Large City — Daniel P. Norton 

Background 

During the fall of 196? Educational Testing Service began to provide consulta- 
tion to the Board of Education of a large city as part of a subcontract for evaluating 
the outcomes of Title I ESEA programs. At tnat time the Board of Education did not 
have formally organized, central evaluation capabilities. Therefore, among other 
services. Educational Testing Service was requested to provide advice which would be 
helpful to the Board of Education when it might attempt to establish a centralized 
evaluation component within Title I and, in general, the funded projects area. 

This report is a summary of the outcomes of our- deliberations. Its contents 
are perhaps of general interest . The management emphasis which we have elected to 
make is somewhat unusual because the specific management viewpoint which has been 
. adopted has not, to my knowledge, previously been strongly supported for its appli- 
cability to the organization of evaluation efforts in local education agencies. Yet 
the approach to be outlined may be relevant to any large city school system which is 
trying to establish or improve the function of a research and evaluation staff. 

Strategic Planning, Management Control and Operational Control 



In a small book titled Planning and Control Systems : A Framework for Analysis , 
Robert N. Anthony reported his thoughts about the broad topical area of planning 
and control systems. His thoughts, I have concluded, are highly appropriate to the 
organization of evaluation efforts in a large city system. To establish the possible 
suitability of Anthony's ideas to this audience I must first define for you 
three critical terms: (1) strategic planning, (2) management control , and (3) op- 
erational control . The viewpoint which has been presented to the contracting school 
system is built upon the use of these terms. 

Anthony himself suggested the following definitions for them: 

Strategic planning is the process of deciding on 
objectives of the/j in these 
objectives, on the resources used to attain these 



1. Boston: Division of Research, Graduate School of Business Administration, 
Harvard University, 1963, ISO pp. 
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objectives, and on the policies that are to govern 
the acquisition, use, and disposition of these 
resources.^ 



Management: control is the process by which managers 
assure that resources are obtained and used effec- 
tively and efficiently in the accomplishment of the 
organization ; s objectives. 



Operational control is the process of assuring that 
specific tasks are carried out effectively and ef- 
ficiently; 



It must be understood that the three activities which have just been 
defined should not be considered to be fully discreet and non-overlapping. On the 
contrary, overlap is sufficiently great that, in particular, strategic planning 
and management control or management control and operational control functions 
sometimes can hardly be separated from each other. 



■ Title I ESEA Evaluation 

i 

To help you understand the applications which are proposed for these 
I terms I must tell you more about the Title I, ESEA, program which requires eval- 
uation. It was budgeted for the expenditure of about $23 million over the previous 
fiscal year. This budget constituted roughly 5 per cent of the national budget 
for Title I and exceeded one half of the budget for the sponsoring state. Forty-two 
projects were undertaken through this funding. Their range spanned a very large 
fraction of the full domain of Title I projects as they exist nationally. Therefore, 
nearly the complete range of evaluation problems which anyone must face exist in 
this single funding, and, it seems safe to say, the issues which exist with regard to 
the evaluation of Title I in, for example, OE or the State Department of Public 
Instruction are essentially identical with those found in the city. 

The administration of the Title I program is undertaken between impinging 
forces for (3.) reorganization for purposes of decentralization which are being . 
applied from above within the Board of Education, and for (2) local rule which are 
being generated at the building and community levels. As part of the decentralization 
effort, which has been initiated from the top of the administrative structure, three 
area superintendents are being installed beneath the office of the General Superin- 




Ibid. , pp. 16 
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tendent of Schools and above the 27 district superintendents who have formerly 
reported to the central administrative offices. Community interest in control of 
schools has risen rapidly within the past several years and has been forcef ully ex- 
pressed as, for example, when popular Title 1 activities were scheduled for elimina- 
tion or removal from some buildings this past fall. At that time community groups 
protested directly to the central administration against termination of Title I activ- 
ities which they judged to be important to their building. These opposing forces 
from above and below upon the director of Title I activities must, it seems, be given- 
serious attention as an evaluation component is installed within the Board of Educa- 
tion. 



Of course, from outside of the Board of Education come demands to provide 
evaluative summaries for the benefit of the State Department of Public Instruction 
and the U. S. Office of Education, The most notable of various external demands 
has been the introduction of the Survey of Compensatory Education which was begun 
last spring. That survey called for the cooperation of central office staff and 
of personnel in 135 buildings within the city. The persuasive and organizational 
aptitudes which we re called for to conduct the Survey of Compensatory Education 
were almost excessive; they are typical of the evaluation demands which exist when 
large program efforts are to be evaluated. Figure 2 summarizes the sources and 
demands for liaison which must be attended to, 

Strategic Planning 



I wish now to call your attention to the basic organizational plan for 

. as Fie/ur<Z 1 

curriculum evaluation and research which has been passed out to yoi^. There is 
hardly anything new in it. It is very brief for a large city, but one must start 
somewhere . 



What I am attempting to present to you is not a drastically different organ- 
izational plan but, instead, a viewpoint toward evaluation in this particular 
setting which, if accepted, has much to say about the selection of personnel for 
positions, priorities to govern their selection, and priorities by which they would 
allocate their time and efforts. To understand this difference you should first 
note that the Director of Evaluation and Research and the Director of Design and 
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Test Development are explicitly recorded as separate positions.. Also, the Director 
of Evaluation and Research is shown in a 11 superior" position to that of the Director 
of Design and Test Development in spite of the fact that the Director of Design and 
Test Development must almost certainly possess a greater level of technical -knowledge 
than the Director of Evaluation and Research. 



To better understand this arrangement, to fully understand the viewpoint which 
is being implied and to envisage the role of the research director you may require 
two additional guidelines. First, the viewpoint I am presenting requires that the * 
question "How do we improve evaluation?" must usually be granted precedence over the 
more immediate question of "How do I evaluate this project?" if evaluation is to be 
fully meaningful. Second, one should • perhaps accept the opinion that^stnfl'^'ln- 
purpose rationale or viewpoint can be provided for directing (managing) evaluation* 



It seems almost impossible to work at the evaluation of compensatory education 
programs without being nearly ovewhelmed by the variety of measurement, logistical 
and other problems which exist. Instruments have been found to be unsuitable to the 
populations; norm groups for the instfetyients which have been employed are generally 
inappropriate. Satisfactory comparison, groups are difficult- to come by. The eval- 
uation of e ^rts^which call for the involvement of community representatives may 
be totally^ suitable instrumentation; even when instrumentation is avail- 

able, the cooperation of different interested and effected parties such as principals, 
teachers and parents is not easy to acquire. In some cities, decentralization and 
the rapid growth of community interest in schools poses another set of exceedingly 
difficult problems to be overcome. The decentralization to which I have referred 
is proceeding along lines laid out by a management consulting firm, but the gen- 
eration of local interest is seemingly without order. 

Directives requesting major changes in program emphasis, as for example from 
•expenditures spread across eligible buildings to expenditures concentrated for 

selected pupils and buildings, make planning for evaluation over years a somewhat 
hazardous undertaking in itself. 



Area superintendencies, as being developed in the schools of the city to which 
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I am relating , will have responsibility for the education of more than 150,000 
students. If area superintendencies are to indeed have considerable autonomy in 
the administration of educational programs within their jurisdiction do they not 
themselves require capabilities for evaluation, and should one not anticipate that 
the administration of Title I funds might someday have to be decentralized to the 
area level? The foregoing and other questions and issues, in almost endless pro- 
fusion, have prompted us to conclude that the most important concerns which would 
demand the attention of a Director of Evaluation and Research in the federal 
project area are concerns which Anthony would subsume under the title of "strategic 
planning," They are also the sort of concerns which a technically proficient 
Director of Design and Test Development would often not find very interesting. 

At least, I should be correct in stating that most specialists in design and test 
development do not wish to spend a very large fraction of their time engaging in 
discussions of strategic issues. 

One reason is perhaps now evident for making a distinction between planning 
to improve evaluation and planning to evaluate a specific activity or undertaking: 
Improvement of Title I evaluation in general requires a great deal of interaction, 
cooperation and planning by many individuals' and agencies, A second set of reasons 
for emphasizing this consideration is explicitly the following: It is usually not 

possible to evaluate a project (singular you will note) and to subsequently provide 
information about it which is both unequivocal and suitably timed for effective 
decision making. By this I also mean to suggest that an organization the size of 
a large city board of education shares problems which are identical in format with 
those to be met by a state department of public instruction and the U. S. Office 
of Education. If, as is admittedly true, 0E has all it can handle in its efforts 
to try to engage in effective evaluation of the outcomes which arise from its 
expenditures, one could hardly expect a large city board of education to perform 
more effectively or, perhaps, to perform as well. Thus the strategic concerns 
which the Office of Education must now attend to for norming, instrument development, 
etc., are shared by large city boards of education; just as 0E cannot solve its 
problems on short order and without long range planning, neither can the board of 
education of a local agency be expected to do so. 
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Concern for the improvement of evaluation in the city must precede, or at 
the very least be kept separate in consideration from , the evaluation of specific 
activities. Thus the role of the Director of Evaluation and Research becomes one 
of (a) a chooser of objectives for evaluation and (b) a planning initiator for the 
organization of evaluation efforts, both of which are strategic concerns. 

Within the framework established by Anthony, the concerns which I have been 
enumerating in this section of my presentation are for strategic planning to deal 
with the following most pressing policy issues : 

1. How to successfully initiate an extensive data 
bank operation with access to city-wide test 
data and provide for the security of data which 
is collected and inserted in it. 

2. The type of liaison which should be initiated 
with area superintendents, district superin- 
tendents, building principals and teachers. 

3. The types of interaction which should be initiated 
with external agencies such as OE, the state 
department of public instruction, other cities 
via conferences, and other collaborative efforts, 

4- How to provide in-service education about evaluation. 

5. Hew to schedule for long-term development of 
evaluation and research capabilities. 

6. How to assign priorities by which to determine 
which issues should be given the greatest attention. 

7. How to allocate money to various evaluation activities. 

8. How to allocate personnel to different evaluation 
activities. 

have beejj 

Ort Figure 2 A recorded the internal and external agencies and individuals of concern 
to the Director of Evaluation and Research when he engages in strategic planning 
efforts. It may be noted that initiation of strategic planning will, as Anthony 
has suggested, involve discussions of strategies for evaluation which, over the 
first year at least, will require a great amount of person-to-person interaction 
with individuals and small groups. From those discussions the director will have 
as his responsibility the task of better identifying and clarifying the efforts 
to be undertaken by his organization. 
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Management Control 



Anthony has said: 

Strategic planning is the process of deciding on objectives 
and policies. It has to do with major decisions with long- 
term consequences c Management control has to do with the 
ongoing operation of the enterprise, within the guide lin es 
established by these policies.^ 

He has firmly emphasized that these processes shade into one another but has 
maintained that the distinctions which can be made between them are meaningful — an 
opinion which I share and which seems particularly appropriate for the Board of 
Education. You have probably noted that the Director of Design and lest Development 
occupies a position in Figure I which allocates to him responsibilities of a very 
broad nature. This is intended to communicate something of the relationship between 
his position and that of other individuals in the organization. One can almost say 
that this individual should identify strongly with management control and also be 
involved in strategic planning and operational control. However, his greatest concern 
would be for those management control issues for which technical design and test 
development considerations must have high priority. Thus the Director of Design and 
Test Development would ideally assume the largest fraction cf responsibilities for 
operational phases of ongoing research and evaluation efforts; the Director of Eval- 
uation and Research would share responsibility for management control functions but 
would concern himself more with establishing role relationships and feasible schedules, 
dealing with budgetary matters and other details which are generally thought to be 
administrative matters rather than research design or development activities. 

Exhibit I is a record of activities which can b° classified under the three 
major framework headings. I have already enumerated to you those issues recorded 
under strategic planning. You will note that the activities recorded under manage- 
ment control are, with one exception, of the nature of elaborations of more specific 
undertakings; they parallel strategic policy issues. Planning for the evaluation 
of activities, the new and last item, is the function most commonly associated with 
a Director of Design and Test Development or, perhaps, a Director of Evaluation and 
Research for a specific evaluation effort. It is a responsibility which would demand 



ft Ibid ,, p. 68. 
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the greatest fraction of time from the Director of Design and Test Development 
and which best distinguishes between the responsibilities of his office and that 
of the Director of Evaluation and Research, 

Operational Control 



You have noted that Exhibit I also provides a summary of activities for 
operational control. Program evaluation and review technique (PERT) is illustra- 
tive of the systematic efforts which have been developed to handle operational 
control problems. Planning memoranda, daily work schedules and similar documents 
are highly appropriate for operational control activities, 
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Clerks, programmers, field service workers 
and research assistants are to carry out operational undertakings; the various 
directors would be responsible for monitoring and controlling those undertakings. 
I assume you fully understand the nature of these efforts. 



A Brief Summary 



One can perhaps understand most of what there is to understand about the term 
"management" by equating it, as Simon has, with decision-making.^ As^such, it is 
hard to say much about the topic I have selected without risking that an audience 
°f- en 5 it not generally, feel they are not hearing anything new. However, I 
feel I must persist in my effort by emphasizing that a curriculum for the prepara- 
tion of researchers in behavioral science or for specialists in research design, or 
even for specialists in evaluation might seldom prepare them to effectively cope 
with the evaluation problems which exist in an agency such as the large city board 
of education. A graduate of such a program who has acquired technical capabilities 
in the areas of test development and research design might be poorly suited to 
engage .in person-to-person interactions with the range of individuals concerned 
with critical aspects of evaluation within the Board of Education. At the same 
time, a person well suited to the latter is likely to be less than fully adequate 
to deal with technical issues. It is not enough, I am suggesting, to secure top 

level staffing for one of these functions without simultaneously having consideration 
for the other. 



(*. Simon, Herbert A., The New Science of Management , New York: Harper & Row 

I960, p. 50. 
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The evaluation of projects in a setting as complexly dynamic as that within 
which large city government-funded programs operate, it seems to me, possesses its 
own strategic issues which should be formally attended to. Therefore, if decisions 
re made, and they must be, to spend or not to spend money on a specific evaluative 
ac ivity, it is important to insure that evaluative efforts will not be one-shot 
affairs excessively keyed to specific curricular undertakings. By emphasizing 
management concerns and providing (hopefully) creditable relationships to a 
rationale such as that presented by Anthony, I have attempted to establish criteria 
for personnel selection, for the order of their selection and for their training 
and assimilation into the evaluation effort being undertaken within the Board of 
Education. This effort may not be fully successful, but, hopefully, will not 
e characterized by lack of concern for collaborative efforts on the behalf of 
long range planning for evaluation nor will it fail to provide some capability 
for adaptation to changes which will inevitably occur within the domain of funded 
projects of the large city for which consultation is being provided. 
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Figure 2. Agencies and Departments with which Liaison 
Must Be Maintained for Purposes -of Planning and Control. 
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Exhibit I 



Examples of Evaluation and Research Activities 
Included Under Framework Headings 



Strategic Planning 


Management Control 


Operational Control 


Setting data bank 
policies 


Planning for initiating 
data banking 


Implementing data bank 
plans 


Setting internal liaison 
policies 


Scheduling liaison 
activities 




Setting policies for 
collaboration 


Deciding on 
collaborative efforts 


Implementing policies 


Setting in-service 
education policies 


Deciding on in-service 
undertakings 


Scheduling in-service 
programs 


Setting long-term 
policies 


Formulating long-term 
programs 


Scheduling long-term 
programs 


Establishing policies 
for the assignment of 
priorities 


Formulating priorities 




Deciding on non-routine 
expenditures 


Deciding on routine 
expenditures 


Controlling 

expenditures 


Setting personnel 
policies 


Formulating personnel 
practices 


Implement ing 
policies 


* 


Planning for the 
evaluation of specific 
curricular activities 


Scheduling and 
monitoring evaluation 
activities (e.g.* 
scoring tests* etc.) 



